A basic idea of the change:
Instead of using channels allocated in the Dial function for both (per peer, and fd consuming) types of rate limiting, we create a rateLimiter object that manages scheduling dials for all outgoing connections. It uses a single lock, and the only internal logic it does is for new dials, check if you can take a token, if so, start the dial in a goroutine, if you cant take a token right now, add the dial to a queue to be started as soon as a token is available. This is functionally equivalent to the existing implementation, except for the fact that it uses vastly fewer goroutines. This simplification in concurrency lets us more confidently use a channel of addresses as input to the dial instead of an array of them. Using a channel of addresses with the current implementation increases the locking complexity (And chance for accidental deadlocks) immensely.

when reviewing this, i recommend looking over the changes in swarm_dial.go, particularly in dialAddrs, then take a look at the dialLimiter code afterwards. It should make more sense that way.

Just a few comments and getting CI to pass -- otherwise LGTM 🐴! Lots of good godocs and thorough tests!